[ROCm][CI] Re-enable ROCm CI & make CI changes as per new CI design - #4361
[ROCm][CI] Re-enable ROCm CI & make CI changes as per new CI design#4361akashveramd wants to merge 7 commits into
Conversation
meta-cla
Bot
added
the
CLA Signed
|
Warning: Unknown label
Please add the new label to .github/pytorch-probot.yml |
|
Seems the bot have an error: #4361 (comment) |
@wwwjn: The bot warning was triggered before commit 232d3f1 added ciflow/rocm to .github/pytorch-probot.yml. That commit is now included in this PR, so the label is properly registered, and the warning won't recur once this is merged. |
Renames integration_test_features_rocm.yaml to integration_test_rocm.yaml and re-enables the ROCm matrix in set-matrix.yaml, which was temporarily disabled while ROCm jobs were red. Key changes: - Adds ciflow/rocm/* tag trigger to integration_test_rocm.yaml so ROCm jobs can be triggered on demand from PRs via tag, in addition to the existing push-to-main trigger. - Restores the ROCm matrix entry in set-matrix.yaml (runner: linux.rocm.gpu.ecosystem.mi350.8, ROCm 10.0, rocm10.0 nightly index). - Fixes the matrix selection logic so that when gpu-arch=rocm is explicitly requested (e.g. from a ciflow/rocm tag run via workflow_call), FULL_MATRIX is used rather than falling through to CUDA-only. Previously IS_MAIN_PUSH/IS_SCHEDULE were the only conditions that included ROCm, which meant tag-triggered workflow_call runs produced an empty matrix. Test Plan: Push a ciflow/rocm/* tag to trigger the ROCm workflow and verify jobs land on linux.rocm.gpu.ecosystem.mi350.8 with ROCm 10.0. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
Adds an AMD GPU (ROCm) row to the Test status table pointing to the integration_test_rocm.yaml workflow. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
Registers ciflow/rocm with pytorch-probot so it can push the ciflow/rocm tag to trigger ROCm CI jobs on demand. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
Renames the workflow from "Integration Tests - Features (ROCm)" to "Integration Tests (ROCm)" and the job from "features" to "rocm", since the workflow runs both features and models test suites. This fixes the confusing "features / 8 GPU Integration ..." prefix shown for both jobs in the PR checks UI. Co-authored-by: AI assistant Co-authored-by: Cursor <cursoragent@cursor.com>
ca621ea to
fa3d1f2
Compare
|
Hi @wwwjn @tianyu-l cc: @akashveramd @jithunnair-amd After making some fixes and running the ROCm CI for this PR, we see equality loss issues with golden numerics files that were developed in #4127 From my understanding, every golden numerics file is generated from CUDA A10G runs, and there is no ROCm CI loss file anymore. We wanted to make sure we were aligned on the design you wanted for this moving forward with ROCm in TorchTitan. Proposed fix: parameterize golden_numerics_path on arch (e.g. llama3_{gpu_arch}.txt) by threading gpu_arch_type into the .format() call in run_tests.py. We can then add new ROCm-specific golden files generated from real ROCm runs leaving every existing _a10g.txt untouched so CUDA is unaffected. If this sounds ok to you, we can add it to this PR. If you have a different idea for this, let us know. |
|
I don't think AMD golden loss should be required. The idea is to guard on any accidental numerical change, so verify it on one hardware would be enough. |
|
@tianyu-l we would also need it somewhere to guard on the end-to-end training numerical changes on AMD. Do you have any suggestions? |
|
@yuankaichen-amd curious why do you need it? E.g. would there be cases where cuda numerics are unchanged, but rocm numerics are changed? |
|
@tianyu-l Yes. It's likely. Although there are unit tests in pytorch, torchao etc to ensure the AMD kernel correctness. It'd be also valuable to have end-to-end training guards. |
|
@yuankaichen-amd OK, then we can host some loss files for rocm. Up to you if you want to host the full spectrum compared with cuda ones. Again, titan devs shouldn't be blocked by them. |
5 participants
This PR covers most of the features that were part of this PR (unmerged & closed) #4039.
This PR makes following CI changes-
ciflow/8gpulabel as a trigger for the CUDA-specific workflows.ciflow/rocmlabel as a trigger for the ROCm-specific workflows.-- This label can be added manually by PR developers wanting to test on ROCm.
-- This label will also be automatically added by the pytorch bot for any PRs that mention ROCm in the title/description.